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This  case  study  is  an  application  of  experimental  design  to  the  test  and 
evaluation  of  surface  radars. 

It  builds  upon  work  done  by  the  Naval  Surface  Warfare  Center,  Corona 
Division. 

We  look  back  into  a  test  that  was  considered  a  landmark  in  M&S-based 
acquisition  and  contrast  the  way  one  objective  was  evaluated  to  the  way 
it  could  have  been  evaluated  with  experimental  design. 

In  the  process,  we  explore  the  attributes  of  a  well  designed  test  and 
demonstrate  the  utility  of  experimental  design  for  planning,  designing, 
executing,  and  analyzing  a  test. 

What  can  we  learn  from  the  data?  What  could  we  have  done 
differently ?  What  can  we  do  different  next  time? 


An  experimental  design  approach  contributes  to 
making  the  test  more  robust,  efficient,  and  cost  effective. 


> 

> 

> 

> 

> 
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Tradeoff  Space 


Scientia  PruUentia  et  Valor 


Adequate 
sample  size 


Effects  on 
performance 


Risk  of  accepting 
a  "bad"  system 


Risk  of  rejecting 
a  "good"  system 


The  Central  Problem  of  Test  is  to  determine  the  true  nature  of  the  system,  in  all  possible  scenarios, 
with  a  finite  number  of  samples  that  yield  valid  conclusions  while  minimizing  the  risk  of  error. 
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Test  Background 
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Test  Operations 


Sacrificed  statistical  confidence 
due  to  test  time  limitations. 

■  a  a  z  ^ 


> 


> 


Had  an  informal  criteria  for 
selecting  test  runs. 


Analysis  limited  to  pass/fail. 


Test  events 

■  80  hrs  total  test  time 

■  18  hrs  of  manned  aircraft  raids 

■  110  electronic  attack  (EA)  techniques 

■  1900  simulated  Anti  Ship  Cruise  Missiles 


One  Objective 

>  Evaluate  detection  performance  for  a  class 
of  threat  representative  targets* 

>  Factors  involved 

■  Three  target  factors  -  A,  B,  F 

■  Two  environmental  factors  -  C,  D 

■  One  system  factor  -  E 


>  Test  strategy 

96  possible  treatments 
30  samples  per  treatment  required 
2880  total  runs  required 
96  hrs  of  test  required-not  enough  time! 
670  runs  conducted 

Assessment  criteria  -  Pass/Fail 


*Other  objectives  are  beyond  the  scope  of  this  brief;  however,  similar 
lessons  apply. 
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Experimental  Design  Guidelines* 
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1.  Formulate  a  clear  statement  of  the  problem. 

2.  Identify  the  proper  responses  to  be  analyzed. 

3.  Identify  the  factors  and  their  levels. 


4.  Choose  an  appropriate  experimental  design. 


Perform  the  test  as  outlined  in  the  test  matrix. 


Perform  the  appropriate  statistical  data  analysis. 
Reach  valid  and  practical  recommendations. 


j 

■>1 

J 


J 


*  Montgomery,  D.  C.  (2013),  Design  and  Analysis  of  Experiments, 8th  ed.,  John  Wiley  &  Sons. 


Experimental  Design  Approach 

Step  1  -  Problem  Definition 
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Performance  Requirements 


Statistical  Parameters 

Confidence  level  (a)  -  0.05 

Effect  to  detect  (8)  -  based  on 
performance  expectations 

Variability  (a)  -  based  on  historical 
data 

S/N  (8/a)  - 1.00  (for  the  case  study) 


We  want  to  evaluate  the  effect  of  five  factors*  on  detection  performance. 


*Six  factors  were  of  interest,  but  data  for  one  factor  was  incomplete;  therefore,  the  study  was  limited  to  five  factors. 
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Experimental  Design  Approach 

Steps  2  &  3  -  Responses  and  Factors 


Scientia  Prudentia  et  Valor  ■ 


Continuous  factors  are  preferred. 


A  single  design  can  be  used  to 
evaluate  multiple  responses. 


Covariates 

Search  Times,  Power,  Sensitivity,  Material  Readiness 


A  (Continuous,  3-levels) 


B  (Categorical,  2-levels) 


C  (Categorical,  2-levels) 


D*  (Categorical,  2-levels) 


E*  (Categorical,  2-levels) 


FA  (Fixed) 


Select  appropriate  factors. 


*  — 


Responses 


Rl-Detection  range 


=^-  R2-Transition-to-track  range 


=^-  R3-Firm  track  range 


^  R4-Engagement  Range 


-J 

f  (Factors) +  <? 

v  7  Noise 


Continuous  responses 
are  preferred. 


*  For  large  designs,  these  are  hard(er)-to-change  factors 
A  Fixed  during  the  original  test  due  to  test  time  limitations 


Experimental  Design  Approach 

Step  4  -  Select  a  Design 
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Some  Experimental  Design  Alternatives* 

Completely  Randomized  Designs;  Model  -  ME  +  2FI;  Power  (1  std.  dev.)  at  a  =  0.05 


Design 

Runs 

Center 

Power  (%) 

VIF 

DOF 

Std.  Error 

Points 

(ME) 

Model 

LOF 

PE 

(FDS=0.8) 

MR-Res  IV 

12 

0 

27-28 

1.1 -7.0 

10 

1 

0 

1.5 

MR-Res  IV 

92 

5 

39-99 

in 

T — 1 

1 

o 

T — 1 

15 

12 

64 

0.7 

9  5-’1 

^V 

16 

0 

- 

1.0 

15 

0 

0 

- 

t — 1 

i 

LO 

> 

CN 

96 

5 

50-98 

1.0 

15 

16 

64 

0.6 

D-Optimal 

21 

0 

54-57 

1.1 

15 

5 

0 

1.0 

2s 

32 

0 

76 

1.0 

15 

16 

0 

0.7 

25  - 

5 

>80 

1.0 

15 

32 

64 

0.5 

2  x2s 

64 

Eve 
1  tl 

iluate  several  design* 

5  and  select  one 

2  x2s 

144 

5 

98-99 

nat  has  good  properties  and  that  is 

4  x  25 

192 

4 

99 

appropriate  for  the  problem. 

Legend: 

ME  -  main  effects 

DOF  -  degrees-of-freedom 

LOF  -  lack-of-fit 

FDS  -  fraction  of  the  design  space 


2FI  -  two  factor  interactions 
VI F-  variance  inflation  factor 
PE  -  pure  error 


*  Other  designs  were  not  explored 
due  to  data  limitations. 
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Experimental  Design  Approach 

Step  4  -  Design  Selection 
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>  Case  I:  2V5 1  fractional  factorial 

■  16  runs 

■  No  degrees  of  freedom  for  estimating  pure  erj, 
lack-of-fit,  or  test  of  significance 


>  Case  II:  25  factor 

■  32  runs 
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Experimental  Design  Approach 

Step  5  -  Execution 
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Run  the  test  as  specified  in 
the  test  matrix. 


:rix  For  Case  III  (25  factori 


Run  Std. 

Order  Order 


Block 


Replicate,  Randomize,  and 
Block  whenever  possible. 


96 

1 

Blkl 

1 

B2 

C2 

D1 

El 

XXX 

35 

2 

Blkl 

-1 

B1 

Cl 

D2 

El 

XXX 

101 

3 

Blkl 

1 

B1 

C2 

D2 

El 

XXX 

46 

4 

Blkl 

1 

B2 

Cl 

D2 

El 

XXX 

32 

5 

Blkl 

1 

B2 

C2 

D1 

El 

XXX 

107 

6 

Blkl 

-1 

B2 

Cl 

D2 

E2 

XXX 

89 

7 

Blkl 

0 

B1 

C2 

D2 

El 

XXX 

56 

n 

Blkl 

1 

B2 

C2 

D1 

El 

XXX 

192 

192 

Blk2 

-1 

B1 

Cl 

D1 

El 

XXX 
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Experimental  Design  Approach 

Step  6  -  Statistical  Analysis  (Case  I) 
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Factor  D  is  not  significant— a  model  is 
suspect.  _ 


Analysis  of  Variance  Table  for  Case  I  (2V5 ^Fractional  Factorial);  p-value  <0.1 


Sum  of 

Mean 

F 

p-value 

Source 

Squares 

df 

Square 

Value 

Prob  >  F 

Model 

167.26 

5 

33.45 

25.70 

<  0.0001 

A- A 

66.59 

1 

66.59 

51.15 

<  0.0001 

B-B 

65.29 

1 

65.29 

50.15 

<  0.0001 

C-C 

26.37 

1 

26.37 

20.26 

0.0011 

D-D 

1.70 

1 

1.70 

1.31 

0.2794* 

BD 

7.32 

1 

7.32 

5.62 

0.039 f 

Residual 

13.02 

10 

1.30 

Cor  Total 

180.28 

15 

Experimental  design  affords  studying 
interactions. 


Ihdiii 


I  □  ■ 


Reduced  Empirical  Model  (Coded  Factors) 

+  x^  +  x2B  -  x3C+  x4D  +  x24BD 

Pred.  R2  =  0.8151  Adeq.  Precision  =  17.23 


9  10  11  12  13  14  IS 


The  empirical  model  is  useful  for  tactical  decision 
aids,  training,  and  performance  assessment. 


Reference  Mechanistic  Model 


r 


R  = 


PtxGxA 
x— - - - x 


c TXF 2 


1  txN 

■X  —  X - 


\l/4 


(S/N)t  (4  xYxk  LtxLxLbslxLalxLs 


p  J 


Experimental  Design  Approach 

Step  6  -  Statistical  Analysis  (Case  I) 
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>  Only  factors  A,  B,  and  C  and  interaction  BD  are  significant;  factor  E  is  dropped 
from  consideration— the  sparsity  of  effects  principle. 

>  A  Res  V  fractional  factorial  design  contains  a  complete  factorial  in  any  subset  of  4 
factors— the  projection  property. 

>  We  can  combine  the  runs  of  fractional  factorials  to  assemble  a  larger  design  (two 
blocks)— sequential  experimentation 


2V5 1  Fractional  Factorial 


Dof  for  evaluation;  model  -  ME  +  2FI 


Model  15 

Residuals  0 

Lack  Of  Fit  0 

Pure  Error  0 

Total  15 


Unable  to  calculate  power 


24  Factorial 


i 

k  Sparsity  of 

E 

"  effects 

A 

§ 

k  Projection 

E 

"  property 

A 

Dof  for  evaluation;  model  -  ME  +  2FI 

Model  10 

Residuals  5 

Lack  Of  Fit  5 

Pure  Error  0 

Total  15 

Power-  37%  (1  std.  dev.); 

89%  (2  std.  dev.) 


Sequential 

experimentation 
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Experimental  Design  Approach 

Step  6  -  Statistical  Analysis  (Case  II) 


Half-Normal  Plot 


& 


Design-Expert®  Software 


Shapiro-Wilk  test 
W-value  =  o  958 
p-value  =  0  347 


Scientia  Prudentia  et  Valor 


Analysis  of  Variance  Table  for  Case  I  (2V5 ^Fractional  Factorial);  p-value  <0.1 


significant 


Factor  D  has  no  significant 
effect o  nj  h  e res  po  n  se^ 


Sum  of 

Mean 

F 

p-value 

Source 

Squares 

df 

Square 

Value 

Prob  >  F 

Model 

627.68 

6 

104.61 

139.00 

<0.0001 

A 

216.38 

1 

216.38 

287.51 

<0.0001 

B 

303.20 

1 

303.20 

402.85 

<0.0001 

C 

94.92 

1 

94.92 

126.11 

<0.0001 

0.27 

1 

0.27 

0.36 

0.5513 

BC 

3.35 

1 

3.35 

4.45 

0.0393 

BE 

9.56 

1 

9.56 

12-U 

0.0007 

Residual 

42.90 

57 

0.75 

Lack  of  Fit 

19.01 

25 

0.76 

1 

0.4742 

Pure  Error 

23.89 

32 

0.75 

_ 

Cor  Total 

670.58 

63 

F-values  con! 

not  significant 


complete  randomization. 


Reduced  Empirical  Model  (Coded  Factors) 

R  =  I  +  x2A  +  x2B  +  x3C  -x4E  +  x23BC  -  x25BE 

Adeq.  Precision  =  39.2 


Adj.  R2  =  0.9293  Pred.  R2  =  0.9193 


Continuous  factors  yield 
response  surfaces; 
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Continuous 


Experimental  Design  Approach 

Step  6  -  Diagnostics  (Case  II) 
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Design 


25  Factorial 
2  Replicates 


Predicted  vs.  Actual 


Normal  Plot  of  Residuals 


Validating  the  data  and  the  statistical  assumptions. 
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Experimental  Design  Approach 

Step  6  -  Statistical  Analysis  (Case 


Factors  D  and  E  are  not 
significant. 


Scientia  Prudentia  et  Valor 


Analysis  of  Variance  Table  for  Case  III  (4  x  25  Factorial  +  Center  Points);  p-value  <  0.1 


Sum  of 

Mean 

F 

p-value 

Source 

Squares 

df 

Square 

Value 

Prob  >  F 

Model 

1524.59 

5 

304.92 

141.81 

<  0.0001 

A 

423.11 

1 

423.11 

196.78 

<0.0001 

B 

636.09 

1 

636.09 

295.83 

<0.0001 

C 

191.10 

1 

191.10 

88.88 

<0.0001 

E 

2.90 

1 

2.90 

1.35 

02467 

BE 

15.04 

1 

15.04 

7.00 

<^0.0089^ 

Curvature 

22.76 

8 

2.84 

1.32 

0.2346<n 

Residual 

382.73 

178 

2.15 

Lack  of  Fit 

52.47 

34 

1.54 

0.67 

0.9112 ^ 

Pure  Error 

330.26 

144 

2.29 

Cor  Total 

1930.08 

191 

significant 


Reduced  Empirical  Model  (Adjusted,  Coded  Factors) 

R  =  I  +  x2A  +  x2B  -x3C  +x5E  +  x25BE 


R2=  0.7899  Adj.  R2  =  0.7843 


Pred.  R2  =  0.7775 


Adeq.  Precision  =  40.9 


Ref:  Design  Expert  8.0.7. 1 
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Experimental  Design  Approach 

Step  6  -  Confirmation 


Save  a  few  runs  for  confirmation. 


itia  Prudentia  et  Valor  h 

Case  I 


Case  II 


Empirical  Error 


□ 

B 

C 

E 

Case  1 

Case  II 

0 

-l 

-l 

-l 

13.8 

5.1 

0 

-1 

-l 

l 

10.9 

0.7 

0 

-1 

l 

-l 

9.0 

3.0 

0 

-l 

l 

l 

0.2 

1.6 

0 

l 

-l 

-l 

7.1 

1.6 

0 

1 

-l 

l 

4.3 

1.4 

0 

l 

l 

-l 

13.6 

0.6 

0 

l 

l 

l 

3.2 

2.0 

Average 

7.8 

2.8 

>  Factor  A  was  a  3-level  factor. 

>  The  designs  for  Case  I  and  Case  II  used  only  the 
high  and  low  settings  (in  blue),  and  not  the 
center  points  (in  red). 

>  The  center  points  were  used  for  confirmation. 

>  The  Empirical  Error  is  the  difference  between 
the  average  (5  runs)  at  the  center  points  and 
the  respective  model  predictions  for  those 
factor  settings. 


" All  models  ore  wrong ,  but  some  are  useful.  " 

George  Box 
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Experimental  Design  Approach 

Test  2 
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1 

II  II 

D  LU 

■  ■ 

M  M 
O  LU 

■ 

M  M 
O  LU 

M  M 
D  LU 

n 

n 

n 

o 

lol 

lol 

„ 

* 

Eri] 

Eri] 

1 

[f] 

C|] 

1 

g 

Eg] 

Eg] 

a 

DC 

g 

Eg] 

Eg] 

1  1 

[  : 

E  ] 

B  ] 

LJ 

r  i 

r  i 

LJ 

Test  1 


Screening  the  factors  resulted  in  50% 
reduction  in  runs  from  test  to  test. 
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Experimental  Design  Approach 

Step  6  -  Statistical  Analysis  (Test  2,  Case  II) 
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Half-Normal  Plot 


Analysis  of  Variance  Table  for  Test  2  (2  x  24  Factorial);  p-value  <  0.1 


Sum  of 

Mean 

F 

p-value 

Source 

Squares 

df 

Square 

Value 

Prob  >  F 

Model 

350.07 

7 

50.01 

310.41 

<  0.0001  significant 

A 

137.99 

1 

137.99 

856.50 

<0.0001 

B 

187.65 

1 

187.65 

1164.74 

<  0.0001 

C 

D  - 

15.83 

1 

Factors  D  now  is  significant. 

k _ d 

AB 

3.23 

1 

zu.uo 

u.  uuuz 

AC 

0.58 

1 

0.58 

3.57 

0.0710 

BC 

0.59 

1 

0.59 

3.67 

0.0674 

Residual 

3.87 

24 

0.16 

Lack  of  fit 

1.68 

8 

0.21 

1.53 

0.2231  not  significant 

Pure  Error 

2.19 

16 

0.14 

Cor  Total 

353.93 

31 

Reduced  Empirical  Model  (Coded  Factors) 

R  =  I  +  xxA  +  x2B  -  x3C  +  x4D  +  x12AB  +  x13AC  -  x23BC 

R2=  0.9891  Adj.  R2  =  0.9859  Pred.  R2  =  0.9806  Adeq.  Precision  =  55.5 


Ref:  Design  Expert  8. 0.7.1 
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Experimental  Design  Approach 

Step  6  -  Statistical  Analysis  (Comparison) 

Scientia  Prudentia  et  Valor 


ANOVA  Table  for  the  split-plot  experiment  (Test  1  vs.  Test  2) 


Source 

dof 

SS 

MS 

F„ 

F 

Significant 

R  (Test) 

1 

16.02 

16.02 

330.90 

10.13 

Significant 

WP  Error 

2 

0.10 

0.05 

0.35 

* 

A 

1 

119.74 

119.74 

872.03 

4.38 

Significant 

B 

1 

147.15 

147.15 

1071.65 

4.38 

Significant 

C 

1 

27.08 

27.08 

197.25 

4.38 

Significant 

AB 

1 

3.45 

3.45 

25.09 

4.38 

Significant 

AC 

1 

0.00 

0.00 

0.03 

4.38 

Not  significant 

BC 

1 

1.16 

1.16 

8.41 

4.38 

Significant 

RA 

1 

0.26 

0.26 

1.89 

4.38 

Not  significant 

RB 

1 

1.73 

1.73 

12.60 

4.38 

Significant 

RC 

1 

2.99 

2.99 

21.77 

4.38 

Significant 

SP  Error 

19 

2.61 

0.14 

* 

* 

* 

31 

322.28 

A  split-plot  design  was  used  to  compare  detection  performance  between  the  tests. 
Factor  R  is  significant— there  is  a  difference  between  the  radar  systems. 
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Summary 
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>  Experimental  design  is  the  integration  of  well  defined  and  structured 
scientific  strategies  forgathering  empirical  knowledge  using  statistical 
methods  for  planning,  designing,  executing,  and  analyzing  a  test. 

>  Experimental  design  provides  a  comprehensive  understanding  of  the 
tradeoffs  in  the  techno-programmatic  domains:  risks,  cost,  and  utility  of 
information. 

>  Experimental  design  can  help  reducing  test  assets,  shortening  the  test 
schedule,  and  providing  more  information  to  the  warfighter  and  decision 
makers. 

>  Experimental  design  adds  rigor  and  discipline  to  T&E. 
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Conclusions 
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What  could  we  have  done  differently ? 


Only  16  runs 
+  center  points 
+  axial  points 
(may  be) 
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